Detecting Human Faces in Color Images

A method is introduced that detects human faces in color images by rst separating skin regions from non skin regions and then locating faces within skin regions A chroma chart is prepared via a training process that shows likelihoods of di erent colors representing the skin Using the chroma chart a color image is trans formed into a gray scale image with the gray value at a pixel showing the likelihood of the pixel representing the skin By segmenting the gray scale image skin regions are separated from nonskin regions Then using the lu minance component of the color image and by template matching faces are located within skin regions Introduction and Background Indexing and retrieval of video images containing human activities require automatic detection of the hu mans and in particular human faces in images Tech niques that recognize faces or analyze facial expressions also require knowledge about the existence of faces in images In this paper a new method for detecting faces in color images is presented Majority of existing face detection techniques rely only on image gray values in spite of the fact that most images generated today are color A small number of techniques use color information to detect faces in images Methods that are based on image gray values detect prede ned image features and use them either in a sys tem that has a learning ability or in a model matching algorithm to detect the faces A method developed by Rowley et al tests for existence of faces of di er ent sizes at each image location using a neural network Although the system is designed for detecting frontal faces in images the network lters can be trained to detect side view faces also An alternative to neural networks is a probabilistic visual learning algorithm designed by Colmenarez and Huang In a system developed by Sung and Poggio rst a model face is generated using an example based learning method Then at each image location a feature vector is com puted between the local image and the model Finally a trained classi er is used to determine whether a face exists at that location or not Many methods rely on feature matching to detect faces A method developed by Yow and Cipolla uses second derivative Gaussian lters elongated at an aspect ratio of to locate horizontal bar like features in images From the bar like features elongated facial features such as the eyes and the mouth are located and by grouping the features and matching their re lations with those of a facial model instances of faces are found By changing the size of the Gaussian lter this method can be used to detect di erent sized faces in images A similar method is described by Cootes and Taylor which instead of bar like features uses statistical features and by matching relations between detected features and those of a model face nds faces in images A method developed by Leung et al uses a graph matching method to nd likely faces from among detected facial features First graphs represent ing candidate faces are generated from detected facial features and then by a random graph matching algo rithm true faces are located from among candidate faces A method that considers a facial pattern a texture pattern and uses texture features to locate faces in an image is described by Dai et al Texture features based on a space gray level dependence SGLD ma trix are used By comparing texture features of a model face with texture features of windows in the image potential faces are located Using model faces and windows of di erent sizes this method can detect di erent sized faces in an image A method that uses only image edges to detect faces is described by Govin daraju In this method second derivative edges are rst determined Then using models of human face curves faces are located within the edges A small number of techniques use color information to detect faces These techniques rst select likely im age regions for faces Then they detect faces in the se lected regions using facial patterns Dai and Nakano isolated and kept orange like regions in the YIQ color space as the skin regions and eliminated the remain ing regions Then they used texture features in gray scale images to identify faces in the skin regions Chen et al prepared a color chart in the HSV color space that represented possible skin colors Then they used the color chart to identify image regions that rep resented the skin Using three face templates they located faces in skin regions through a fuzzy pattern matching algorithm Face templates of di erent sizes were used to detect di erent sized faces in an image Sobottka and Pitas detected skin regions in an im age using the hue and saturation and selected regions that were elliptic as faces Miyake et al used a chart of skin colors in the Luv color space to detect skin regions in an image They then located round regions from among obtained regions to detect faces Schiele and Waibel used the red and green components of the skin color in a large number of images to construct a histogram They then used the histogram to com pute the probability that a particular combination of red and green belonged to the skin and used the prob abilities to classify image pixels to skin and nonskin regions Crowley and Berard used this method to track faces in video images In the method proposed here information about skin colors is used rst to nd likely image regions where a face may exist Then information about fa cial patterns is used to determine instances of faces in such regions The di erences between the proposed and previous face detection methods can be summa rized as follows Computations are performed in the CIE LAB color space Colors in CIE LAB space are more uniformly spaced than colors in RGB or HSV spaces enabling use of a xed color dis tance in decision making over a wide spectrum Only chroma a and b components in CIE LAB is used to separate skin from nonskin regions Be cause of the smooth and curved shape of faces re ected light intensity across a face varies con siderably therefore luminance component of the color can mislead the skin detection process Chroma across a face however remains relatively unchanged and can be reliably used to detect skin regions After skin regions are detected the lu minance component of the colors is used to detect facial patterns Rather than classifying pixels to skin and nonskin regions a weight is assigned to each pixel showing the likelihood of the pixel belonging to the skin The weights are obtained from a chroma chart that is prepared through a training process Using the chroma chart a color image is trans formed into a gray scale image with the gray value at a pixel showing the likelihood of the pixel belonging to the skin This gray scale image is then segmented into skin and nonskin regions The proposed method is based on a bottom up approach therefore it can nd faces of di erent sizes and orientations in an image This is in con trast to top down methods that search for faces of prespeci ed sizes and orientations The proposed face detection method consists of the following steps Determine a chroma chart using the ab compo nents of colors in the CIE LAB color space such that the value at entry a b shows the like lihood that chroma a b represents the skin Transform a color image into a gray scale image with the gray value at a pixel showing the likeli hood of the pixel belonging to the skin Determine skin regions in the obtained gray scale image by image segmentation Detect facial features in skin regions and hypoth esize faces Verify existence of hypothesized faces by tem plate matching These steps are described in more detail next Preparing the Chroma Chart A chroma chart is used to train a vision system to distinguish skin from nonskin colors Color charts have been used previously with considerable success Previous color charts though have been binary colors have been grouped to either skin or nonskin In our model any color can represent the skin but with a di erent likelihood Our color chart will have two components repre senting the a and b values in the CIE LAB color coordinate system Therefore we refer to our color chart as the chroma chart As mentioned earlier we do not use the luminance component of a color to prepare this chart because luminance may vary considerably across a person s face and cannot be a reliable measure to separate skin from nonskin regions Chroma on the other hand remains relatively unchanged across a skin region and can be reliably used to separate skin from nonskin regions Chroma has been used e ectively to segment color images before Our chroma chart is obtained by mapping the ab values in the CIE LAB color space to discrete array entries Entries of the chart are initially set to To train the system skin color samples are taken from a large number of images the colors are transformed into CIE LAB color coordinates the chromas of the colors are mapped to the array entries and entries are changed to This mapping involves transforming oating point values for a in the range to discrete array indices in the range and trans forming oating point values for b in the range to discrete array indices in the range A pair of ab values obtained from the color at an im age pixel will therefore correspond to an entry in a chroma chart To reduce the e ect of noise in the samples instead of using the color at a single pixel we use the aver age color of a window centered at the pixel To obtain the average color rst the average red aver age green and average blue in the window are determined The average red green and blue values are transformed into Lab coordinates and the ab com ponents are used to identify the entry in the chroma chart The entry of the chart is then changed to We used twenty three hundred skin samples in eighty im ages to construct our chart As the number of samples increases the chart becomes denser Skin colors ll only a part of the chroma chart Since adjacent entries in the chart show very similar colors it is inconceiv able that while some entries in a small neighborhood belong to the skin other entries in the same neigh borhood don t Noting that as more skin samples are taken more entries in the chart become lled we see the need for lling the small gaps in the chroma chart The small gaps in the chroma chart can be lled in a variety of ways We however notice that simply lling the gaps is not su cient There are certain ar eas of the chroma chart that represent the skin with a higher likelihood than other areas as evidenced by denser samples There are entries that are not very likely to belong to the skin because only sparse sam ples are available Some of these samples could be due to noise or re ections from skin specularity We need a chroma chart that contains likelihoods that are pro portional to the local densities of the skin samples To obtain such a chroma chart we center a D Gaussian of height at each sample point and record at each chart entry the sum of the obtained Gaussians That is if N skin samples f ai bi i Ng are available we let the likelihood that the chroma at chart entry a b represent the skin be proportional to